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ABSTRACT 


The Navy College Program for Afloat College Education (NCPACE) is one of the main 
components of the United States Navy’s Voluntary Education (VOLED) program, 
offering college courses and remedial academic skill modules to sailors on sea duty or 


stationed in remote locations. 


This thesis predicts the likelihood of NCPACE course completion by course and 
individual participant characteristics using a logistic regression model. We found that 
participants who take distance learning-based and mathematics courses have lower 
predicted odds of succeeding, while participants with higher Armed Forces Qualification 
Test (AFQT) scores and levels of education have higher predicted odds of succeeding. 


Some variation was noticed between unit vessel types and school. 


This thesis also evaluates the likelihood of attempting and successfully 
completing a subsequent course conditional on the outcome of the first course. 
Successful completion of the first course is positively associated with an enrollment in a 


subsequent course in addition to the successfully completion of that course. 


Lastly, this thesis examines promotion, extension and reenlistment outcomes for 
first-term NCPACE participants with 48-month contracts. We found that those who 
successfully completed at least one course are predicted to be more likely to promote to 


ES and are predicted to be slightly less likely to reenlist in the Navy. 
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EXECUTIVE SUMMARY 


Department of Defense (DoD) Directive 1322.08E states that it is DoD policy to establish 
and maintain voluntary education (VOLED) programs that are available to all active duty 
personnel, regardless of duty location. WOLED programs are defined as continuing, 
adult, or postsecondary education programs that service members elect to participate in 
during their off-duty time and are directed to provide educational opportunities 


comparable to those that are available to civilians. 


The Navy College Program for Afloat College Education (NCPACE) is one of the 
main components of the United States Navy’s VOLED program. NCPACE offers 
college courses and remedial academic skill modules to sailors on sea duty. These 
courses are delivered through electronic methods (Technology PACE) or traditional 
classroom instruction (Instructor PACE). Courses are tuition-free, and sailors pay only 
costs associated with college coursework educational materials. All undergraduate 
courses are from institutions affiliated with the Service members Opportunity College 
degree program for the Navy, ensuring sailors the opportunity to transfer credits and 


complete degrees. 


A recent study indicates a significant rise in the number and percentage of sailors 
enrolling in distance education courses through the Navy’s Tuition Assistance (TA) 
program between 1995 and 2008. This growth in distance education enrollments has 
been attributed to the increase in operational tempo since September 2001. Distance 
learning course completion rates are reported as lagging behind traditional course 
completion rates for all years considered, with Physical Science courses experiencing the 


biggest differences in completion rates between instruction methods (Woosely, 2009). 


Though the study referenced above makes an inferential connection between 
distance learning and operational tempo events, it and other evaluations of military- 
sponsored continuing education programs done in the past do not directly address how 
students perform academically in operational environments. This thesis attempts to fill 


this gap by evaluating the effectiveness of the NCPACE component of VOLED. 


XV 


NCPACE is a good candidate for addressing the effectiveness of distance education and 
academic success in operational environments in general, as the program’s students are 
all on sea duty and the program has a distance education component. Identifying the 
factors associated with successful NCPACE course completions provides information 


necessary to make VOLED programs more effective as a whole. 


This thesis predicts the likelihood of NCPACE course completions by course and 
individual participant characteristics using a logistic regression model. We found that 
participants who take distance learning-based and mathematics courses have lower 
predicted odds of succeeding, while higher Armed Forces Qualification Test (AFQT) 
scores and levels of education resulted in higher predicted odds of succeeding. Some 


variation was noticed between unit vessel types and school. 


This thesis also evaluates the likelihood of attempting and successfully 
completing a subsequent course with a consideration given to the outcome of the first 
course. Successful completion of the first course is positively associated with 


enrollments in subsequent courses and in successfully completing subsequent ones. 


Lastly, this thesis examines promotion, extension and reenlistment outcomes for 
first-term NCPACE participants with 48-month contracts. We found that those who 
successfully completed at least one course are predicted to be more likely to promote to 


ES and are predicted to be slightly less likely to reenlist in the Navy. 
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I. INTRODUCTION 


A. BACKGROUND 


Department of Defense (DoD) Directive 1322.08E states that it is DoD policy to 
establish and maintain voluntary education (VOLED) programs that are available to all 
active duty personnel, regardless of duty location. VOLED programs are defined as 
continuing, adult, or postsecondary education programs that service members elect to 
participate in during their off-duty time and which are directed to provide educational 


opportunities comparable to those that are available to civilians. 


The Navy College Program for Afloat College Education (NCPACE) is one of the 
main components of the United States Navy’s VOLED program. NCPACE offers 
college courses and remedial academic skill modules to sailors on sea duty or in 
designated geographically isolated installations. What makes this program important is 
that it is specifically geared towards Service members who would otherwise have 
difficulty gaining access to continuing education resources due to their physical location 
and less-than-predictable work schedules. These courses are delivered through electronic 
methods (Technology PACE) or traditional classroom instruction (Instructor PACE). 
Courses are tuition-free, and sailors pay only the costs associated with college 
coursework educational materials. All undergraduate courses are from institutions that 
are accredited by an accrediting body recognized by the Department of Education and 
affiliated with the Servicemembers Opportunity College degree program for the Navy, 
ensuring sailors the opportunity to transfer credits and complete degrees (OPNAVINST 
1560.9A, 2008). 


Over time, numerous Navy administrative messages (NAVADMINs) have 
emphasized the importance of an educated Navy force, highlighted the Navy’s 
commitment to providing educational opportunities for sailors, and stressed the difficulty 
of successfully completing courses and degrees alongside performing official Navy 


duties. This is well illustrated by the Navy’s decision in 2005 to require an associate’s 


degree for E8 selection in fiscal year 2011 and its subsequent decision to rescind this 


requirement in 2007 (NAVADMIN 203/05, 2005; NAVADMIN 150/07, 2007). 


Several of these messages specifically addressed changes to NCPACE policies 
and guidelines. In May 2004, NAVADMIN 109/04 announced an interim NCPACE 
policy that aimed to make the program more cost-effective and yield higher success rates 
among participants. The policy announced that participants were required to receive 
written approval from their commands and limited enrollment to two courses per term, 
with first time Technology PACE students limited to one course. The message stipulated 
minimum course enrollment levels, stipulated ratios of computers to students, and called 


for command leadership to play a more active role in ensuring NCPACE successes. 


In June 2007, NAVADMIN 161/07 announced additional provisions for sailors 
enrolling in NCPACE courses. This message stated that participants were required to 
have passed their most recent advancement exam (or have approval from their 
commanding officer), disallowed enlisted members with fewer than twenty years of 
service who had less than one year remaining on their agreement, and required education 


plans for sailors who have completed their fifth course. 


In February 2008, NAVADMIN 042/08 rescinded the restriction for enlisted 
sailors with less than one year of service left and reiterated guidelines for command 
involvement and the requirement for sailors who have taken more than five courses to 
commit to their education plan. It placed additional restrictions against taking courses 
towards degrees already earned, with exceptions granted for sailors who change their 


designator or rating. 


In March 2010, NAVADMIN 105/10 added additional requirements for NCPACE 
participants and reemphasized command leadership involvement. It required the 
aforementioned individual education plan for all enrollees, regardless of number of 
classes completed, and restricted enrollment to classes that lead to a degree. A 
prerequisite of completing one year at the first permanent duty station before seeking 


approval for NCPACE courses was added. It directly addressed the issue of course 


completions by calling for those having trouble with their coursework to talk to their 


academic advisors and command leaders before dropping a course. 


A recent study indicates a significant rise in the number and percentage of sailors 
enrolling in distance education courses through the Navy’s Tuition Assistance (TA) 
program between 1995 and 2008. This growth in distance education enrollments has 
been attributed to the increase in operational tempo since September 2001. Distance 
learning course completion rates are reported as lagging behind traditional course 
completion rates for all years considered, with Physical Science courses experiencing the 


biggest differences in completion rates between instruction methods (Woosely, 2009). 


Though the study referenced above makes an inferential connection between 
distance learning and operational tempo events, it and other evaluations of military- 
sponsored continuing education programs done in the past do not directly address how 
students perform academically in operational environments. This thesis attempts to fill 
that gap by evaluating the effectiveness of the NCPACE component of VOLED. 
NCPACE is a good candidate for addressing the effectiveness of distance education and 
academic success in operational environments in general, as the program’s students are 
all on sea duty and the program has a distance education component. Identifying the 
factors associated with successful NCPACE course completions will provide the 


information needed to make VOLED programs more effective as a whole. 


B. OBJECTIVES 


The primary objective of this thesis is to determine factors associated with 
successful NCPACE course completions. Both participant and course characteristics will 
be evaluated. It will also attempt to quantify the impact a failed course has on the sailor’s 
desire to take additional college courses and factor changes made to NCPACE policies 
and guidelines over time. This thesis characterizes what leads to a sailor’s ability to 
succeed in continuing education programs in operational environments and to serve as an 
aid in screening and counseling VOLED participants. This thesis also looks for trends in 
program participation rates and passing rates over time. The course completion analysis 


will be done at the sailor level with respect to the first course taken. 
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The secondary objective of this thesis is to determine if the outcomes of NCPACE 


courses have an effect on promotion and retention. 


1. Literature Review 
a. Garcia and Joy (1998) 


Prior analyses of DoD VOLED programs focused on Tuition Assistance 
(TA) programs and were primarily concerned with how TA usage impacted promotion 
and retention. Garcia and Joy (1998) published one of the few studies that specifically 
identified factors associated with succeeding in NCPACE in addition to addressing the 
effectiveness of Navy VOLED as a whole. Their model distinguished the following 
sailor and course characteristics as factors that caused significant differences in NCPACE 


college level course completion rates: 


e Higher-ranked sailors had higher completion rates: 71 percent of 
courses taken by El and E2s were successfully completed, whereas 81 
percent of courses taken by E3 through E9s were successfully 


completed. 


e Instructor-based classes had higher completion rates: courses taken 
through Technology PACE had a lower completion rate than those 
taken through Instructor PACE (77 percent of Technology-based 
classes were successfully completed versus 84 percent of Instructor- 


based classes). 


e Mathematics classes had lower completion rates: 67 percent of 
mathematics classes were successfully completed, while 81 percent of 


non-mathematics classes were successfully completed. 


e First attempts are more successful than second attempts: 80 percent of 
first courses were successfully completed, whereas only 67 percent of 


second courses were successfully completed. 


e Submariners had higher completion rates: completion rates aboard 
submarines were at least 10 percentage points higher than completion 


rates aboard other ship types. 


e Orientation participation, AFQT score, and previous college 
experience were also associated as factors that affected completion 


rates. 


e For lower division courses, NCPACE success rates were lower than 
success rates associated with TA: Garcia and Joy (1998) hypothesized 
that completion rates were lower for NCPACE courses because TA 
users were required to pay back any assistance used towards a failed 


course and were typically in less demanding work environments. 


These completion rate estimates were based on NCPACE course data 
dating between July 1995 and May 1996. Garcia and Joy (1998) did the regression 
analysis in two stages to correct for a potential selection bias that may exist between 
Instructor PACE and Technology PACE courses, as Technology courses have 
characteristics that may make them more demanding. The first regression modeled the 
probability of receiving instruction electronically. The second regression modeled the 
probability of successfully completing a PACE course using a correction factor derived 


from the first model. 


Both the promotion and retention regression models were estimated in two 
stages in the same manner as the NCPACE model. This was done to correct selection 
biases in which may exist as a result of VOLED participants having higher than average 
motivation. The first stage modeled VOLED participation using involvement in 
academic counseling as an instrumental variable. This was used to calculate a correction 
factor for the promotion and retention models. Other variables considered in the 
promotion and retention models were AFQT score, sea experience, age, marital status, 


and rating group. 


A positive relationship was found between college credits earned through 
VOLED as a whole and promotion to E5 in five years. Promotion estimates were based 


eS) 


on a single group of active duty enlisted sailors who accessed in the last two quarters of 
fiscal year 1992 and served in the Navy at least five years. The ordinal nature of 
paygrade was captured by using an ordered probit model. The extent of vacancies in the 


rating was also considered in the promotion model. 


Regression estimates of retention were based on a single fiscal year 1992 
cohort of active duty enlisted sailors with four-year contracts. A positive relationship 
was found between college credits earned through VOLED and reenlistments. 
Reenlistment bonus information was also captured in the retention model. One of the 
shortcomings of this analysis is that less than a year’s worth of NCPACE course data was 
used to evaluate the effectiveness of that component and only a single cohort was 
evaluated for the entire VOLED program. In addition, the two-stage estimation methods 
in this study have limitations because the instrumental variable chosen (enrolling in 
distance learning courses, participating in academic counseling) and the respective 
outcome (successfully completing a course, promoting to E5 in five years) is likely 


correlated with the unobservable factor of individual motivation. 


b. Buddin and Kapur (2002) 


Buddin and Kapur estimated the effects of TA participation on first-term 
Navy and Marine Corps reenlistments. It criticized the retention model in Garcia and Joy 
(1998) for including sailors who separate from the Navy before the end of their term, as 
these sailors do not have the same chance to utilize the TA program as sailors who stay 
the length of their contract. They found that when members who separated early were 
excluded from the sample, TA users were less likely to reenlist in the Navy by 9 


percentage points. 


This study did not factor TA program performance, and was limited to 
sailors and marines who completed their first term in fiscal year (FY) 1997 and the first 


half of FY 1998. 


C Sticha et al. (2003) 


This study evaluated the Army Continuing Education System (ACES). 
Similar to the Garcia and Joy study, reenlistment and promotion of first-term Active Duty 


Army enlisted soldiers were used as measures of effectiveness. 


Sticha et al. estimated that TA participation resulted in a 7.6 percent 
increase in reenlistment rates. The reenlistment analysis looked at soldiers who began 
and completed a three- or four-year contract between October 1995 and September 2001 
and completed enough service to become eligible for reenlistment. Soldiers stationed at 
installations where the Educational Management Information System (EDMIS) database 
was not operational were excluded. The authors acknowledged this resulted in a biased 
sample, as those stationed at bases in which EDMIS is operational do not make up a 
random sample of all first-time enlistees. Factors considered in the reenlistment model 
were paygrade, military specialty, reenlistment bonus information, Career Management 
Field (CMF), race, sex, level of education and marital status in addition to participation in 
TA and Functional Academic Skills Training (FAST) components of ACES. A bivariate 
probit model which simultaneously estimated participation in TA and reenlistment was 


used to control for selection bias. 


Performance and promotion analysis was based on data from the NCO21 
Validation Project, which collected information about ACES participation from Non- 
commissioned officers (NCOs). Sticha et al. found that participation in TA, MOSIT 
(Military Occupation Specialty Improvement Training) courses, and NCO Leader Skill 
Enhancement courses are positively associated with promotion and performance. One of 
the shortcomings of the database used for the performance and promotion analysis is that 


it had fewer than 2,000 soldiers and the course data relied on self-reported information. 


d. Mehay and Pema (2008) 


Mehay and Pema analyzed the impact of the Navy’s TA program on the 
retention and promotion of first-term Navy enlisted personnel. Their approach to the 
selection bias problem differs from the two previously mentioned studies in that they 
used a sample consisting solely of TA participants, assuming that sailors who choose to 
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participate in TA have similar motivations and that those who withdraw were forced to 
do so by external circumstances such as military deployments and job duties. Their 
choices of control variables in their retention and promotion models were similar to those 
in previous studies: they used race, ethnicity, sex, education, AFQT scores, marital 
status, and dependents. Like the ACES study, Mehay and Pema considered multiple 


cohorts instead of a single one. 


They noted that women used TA at twice the rate of men and had higher 
rates of successful course completion. They also found that minority groups were more 
likely to use TA than whites. Their study found that sailors who use the TA program to 
enroll in college classes have a significantly higher probability of reenlistment and of 
promotion to both E4 and E5 than those who participate but do not complete their 
courses. They ran models with the full sample and compared them to the models that 
included only TA participants and estimated the self-selection bias difference between 
the full sample and the TA participants. One potential problem is the assumption that 
sailors withdraw due to external circumstances, which may be too strong. The descriptive 
statistics for the restricted sample demonstrate that course completers have higher AFQT 
scores and also indicate that that sailors who attained higher levels of education had more 


success in completing courses. 


C; SCOPE, LIMITATIONS AND ASSUMPTIONS 


This study is restricted to Navy enlisted personnel who participated in the 
NCPACE program between fiscal years 1995 and 2008. We excluded warrant and 
commissioned officers from the statistical models in order to obtain a more homogenous 


sample. 


NCPACE courses were only available to commands with a sea duty unit 
identification code (UIC) of type 4, some remote locations, and to sailors supporting 
contingency operations (OPNAVINST 1560.9A). Sailors were also subjected to 
additional requirements, which evolved over time. Because data that determines a 
sailor’s eligibility to participate in NCPACE at a given time was not available, we did not 


consider program participation rates. 


The promotion, extension, and reenlistment models are limited in the sense that 
we did not obtain data which determines a sailor’s eligibility to promote (such as 
promotion board data) or motivators for extension and reenlistment decisions (such as 


benefits or bonuses for these actions). 


D. COURSE OF STUDY 


Chapter II provides a description of the data sources and sets, some descriptive 
statistics and univariate analyses of course completion rate by variables of interest. 
Chapter III provides the variables, assumptions and results of the course completion, 
effects of failed courses, promotion, and reenlistment models. Chapter IV provides 


recommendations and conclusions. A list of references is provided at the end. 
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Il. DATA AND DESCRIPTIVE STATISTICS 


A. DATA SOURCES AND SETS 
1. Navy College Management Information System (NCMIS) 


NCMIS was the source of NCPACE participation data. This data set is an 
accumulation of course rosters for NCPACE classes administered between fiscal years 
1995 and 2009. These data provide extensive information on both students and courses. 


More specifically, the NCMIS data set: 


e identifies course enrollments and results by individual sailor, making it 


possible to track individuals who have taken more than one course; 


e provides demographic information about the sailor (race, ethnicity, sex), 
as well as characteristics about him or her (rank, rating, education level, 


AFQT score) at the time the request to take the course was made; 


e describes course attributes, such as the title, level (lower division, upper 


division, graduate), credit hours and delivery method; 


e gives information about the school, the command, the NCPACE office 
that sponsored the course, the fiscal year that the course request was 
made, and some information about the associated contract (contract 


number and costs). 


The course start date was used to determine the fiscal year the course was taken. 
Records corresponding to courses beginning in fiscal year 2009 were removed from our 


sample because the set of fiscal year 2009 courses appeared to be incomplete. 


2s Defense Manpower Data Center (DMDC) 


DMDC provided sailor-level information on Active Duty accessions, strength, 


reenlistments and separations between October 1994 and December 2009. 


The accession data set contained accession dates, AFQT score percentile, and a 


prior military service indicator at the time of accession. 
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The strength data set provided quarterly snapshots of the Active Duty force and 
included the following: branch of service, paygrade and date of rank, unit identification 
code, time in service, primary occupation, sex, race, ethnicity, marital status, current 


enlistment date, education level and enlistment contract length.. 


The reenlistment and separation data set contains the date of the action. The 


separation data contains separation codes in addition to the action date. 


B. DESCRIPTIVE STATISTICS 
1. Overall NCPACE Course Enrollments 


NCMIS reported 262,479 NCPACE course records between fiscal year 1995 and 
fiscal year 2009. This section considers 260,803 records, which corresponded to courses 


with valid course start dates between fiscal year 1995 and fiscal year 2008. 


The number of overall NCPACE courses taken varied by fiscal year, with an 
increase between 1995 and 2001, a decrease between 2001 and 2005, and an increase 
between 2005 and 2006 (Figure 1). The proportion of Technology courses taken varied 
between 32.8 percent (fiscal year 2004) and 51.8 percent (fiscal year 2007). The 
decrease in overall participation between 2001 and 2005 may be explained by increases 
in sailor duties as a result of the events that occurred on September 2001. It is likely not 
explained by changes in the number of eligible participants, as the changes in the number 
of courses taken did not correspond to changes in Navy enlisted end strength figures 


during this time period (Figure 2). 
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Figure 1. . NCPACE Courses Taken by Fiscal Year (FY1995—FY2008). 
Source: NCMIS. 
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Figure 2. Navy Enlisted End Strength by Fiscal Year (FY 1995—FY 2008). 
Source: DMDC. 


The proportion of successful course completions generally hovered around 70 
percent, with a decrease in completion rates that began in 1999 that lasted through 2003 
(Figure 3; note that the vertical scale starts at 50 percent).! Excluding Invalid/Unknown 





! Succeeded was defined as attaining a course grade of A, B, B+, C, or S. Did Not Succeed was 
defined as attaining a course grade of D, DP, F, FN, I, N, NC, NF, NS, W, WA, WD, WF, WP or WX. 
Invalid/Unknown course grades were not factored into the denominator of the percentages but are shown in 
the graph. 
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observations, the highest course completion rate was experienced in 2008; however, this 
year also experienced the highest number of invalid and unknown course grades. 
Changes in course completion rates could be the result of NAVADMIN 109/04, 
NAVADMIN 161/07 and NAVADMIN 042/08. 
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Figure 3. | NCPACE Success Rate by Fiscal Year (FY1995—FY2008). 
Source: NCMIS. 


An overwhelming majority of NCPACE courses were taken by enlisted 
personnel. Between fiscal years 1995 and 2008, 97 percent of course attempts were made 
by enlisted personnel, with 82.5 percent of attempts made by sailors with paygrades of E3 
through E6 (Figure 4). The remainder of this thesis is restricted to Navy enlisted 


personnel, with the assumption that this population is homogenous in motivation. 
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Figure 4. Courses taken by Pay Grade at the Time of the Course 
(FY 1995—FY2008). Source: NCMIS. 
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2. First-time NCPACE Enrollments 


A considerable number of participants have taken more than one course. Figure 5 
shows the number of courses taken versus the number of participants by fiscal year. As a 
key objective of this thesis is to serve as a tool for screening participants, we have chosen 


to evaluate the first course taken for each participant. We assume that participants who 








make second attempts and beyond can be screened based on previous performance. 
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Figure 5. | NCPACE Courses versus Students by Fiscal Year (FY 1995—FY2008). 
Source: NCMIS. 


A data set containing first-time NCPACE enrollees with valid enlisted paygrades 
was extracted from the NCMIS course roster. We chose the record with the earliest start 
date for each individual enrollee and validated that enrollment record against the 
corresponding DMDC quarterly strength file. From this data set, only records with valid 
course grades were retained. The resulting data set contains 112,914 observations. First 
course attempts and success rates experience trends similar to those of overall NCPACE 
course enrollment levels and success rates. The remainder of this section examines how 


first course success rates vary by course and sailor factors considered individually. 


The rate of successful first course completions varied by course attributes. Three- 
fourths (75.9 percent) of Instructor-based course participants succeeded, whereas only 
about half (48.1 percent) of Technology-based course participants succeeded (Figure 6). 


Different success rates were also observed across course subjects (Figure 7). Those who 
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took mathematics (58.3 percent) and physical and other science (50.3 percent) courses 
did not succeed at rates comparable to those who took humanities, social science and 
business courses (67.1 percent). In addition, the success rate varied across the schools 
that administered the course (Figure 8). 75.4 percent of first time participants who took 
courses through Central Texas College succeeded, while only 37.6 percent of participants 


who took courses through Richland College succeeded. 
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Figure 6. | NCPACE First Course Success Rate by Course Delivery Method 
(FY1995-FY2008). Source: NCMIS. 
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Figure 7. | NCPACE First Course Success Rate by Course Subject (FY 1995—FY 2008). 


Source: NCMIS.2 
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Figure 8. | NCPACE First Course Success Rate by School (FY 1995—FY 2008). 


Source: NCMIS. 


The rate of successful first course completions also varied by attributes of the 
sailor taking the course. Sailors who completed at least some college were most 


successful, whereas those who held an alternative high school credential were least 


2 Course categories were determined by a combination of course subject and title reported by NCMIS. 
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successful (Figure 9). Higher ranked sailors succeeded at higher rates than lower ranked 


sailors (Figure 10). Also, there is some variance in success by rating (Figure 11) and unit 


type (Figure 12). 
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NCPACE First Course Success Rate by Education Level (FY1995—FY2008). 
Source: NCMIS. 


Figure 9. 
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NCPACE First Course Success Rate by Rank (FY1995—FY2008). 
Source: NCMIS. 


Figure 10. 
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Figure 11. _NCPACE First Course Success Rate by Rating Group (FY1995—FY2008). 
Source: NCMIS and Navy Enlisted Manpower and Personnel Classifications and 
Occupational Standards (Navy Enlisted Manpower and Personnel Classifications and 
Occupational Standards, 2010).3 
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Figure 12. .NCPACE First Course Success Rate by Unit Type (FY 1995—FY 2008). 


Source: NCMIS and Naval Vessel Register (NVR).4 


3 Chapter III.A.2 describes the categorization of ratings in detail. 
a Chapter III.A.2 describes the categorization of unit types in detail. 
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Some variation in success rates were observed between the following groups: 
e AFQT Category I-II[A (66.3 percent) and AFQT Category IIIB-V (61.3 
percent success rate). 


e Married (68.7 percent) and unmarried (62.5 percent) sailors. 


Little variation in success rates were observed in the following sailor 


characteristics: 


e Sex: Females (65.3 percent) and males (64.9 percent). 


e Race: American Indian/Alaskan Natives (64.1 percent), Asians (66.8 
percent), Blacks (64 percent), Pacific Islanders (66.9 percent), Whites 
(65.9 percent) and Other (63.3 percent). 


e Ethnicity: Hispanics (65 percent) and non-Hispanics (64.3 percent). 
3. Impact of Failing the First Course 


This study also aims to quantify the impact of failing the first course. We merged 
information about subsequent NCPACE course attempts to the data set of first-time 
enrollments described in the previous section in order to: 

e determine if sailors who did not succeed at their first attempt make 
subsequent attempts. 


e evaluate the performance of sailors who did not succeed at their first 
attempt who make subsequent attempts. 


For each participant, we defined the subsequent attempt as the course with the 
earliest reported start date occurring after the first course’s reported completion date. 
Records with second courses that did not have valid enlisted paygrades or course grades 
were not considered. This resulting data set contained 112,444 observations, with 50,474 


instances of subsequent course attempts. 


Successful completions of first courses seem to be positively associated with 
enrollments in subsequent courses. Overall, 44.9 percent of NCPACE participants made 
a subsequent attempt after the completion of their first course. Less than one-third (32.1 
percent) of the participants who did not successfully complete their first course made a 
subsequent attempt, whereas about half (51.8 percent) of those who succeeded made a 
subsequent attempt (Figure 13). 
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Figure 13. Rate of Subsequent NCPACE Course Attempts by Result of First Course 
(FY1995-FY2008). Source: NCMIS. 


There is also some indication that successfully completing a first course is 
associated with successfully completing subsequent ones. While the overall subsequent 
course completion rate of 66.9 percent is comparable to the first course completion rate 
of 65 percent, there is a noticeable difference by first course results. Seventy-five percent 
of participants who successfully completed their first course and taken a subsequent 
course passed the subsequent course, whereas only 42 percent of participants who did not 
successfully complete their first course and taken a subsequent course passed the 


subsequent course (Figure 14). 
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Figure 14. Success Rate of Subsequent NCPACE Courses by Result of First Course 
(FY1995-FY2008). Source: NCMIS. 
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4. Impact on Promotion and Retention 


Lastly, this study examines if NCPACE course outcomes have promotion and 
retention effects on first-term sailors with four-year contracts and non-prior service 
accessions occurring between fiscal years 1995 and 2004. We created the following data 


sets using the source data: 


e Using the NCMIS course roster, we consolidated each participant’s course 
history into a single record, which reported the course start dates 
associated with the first successful and unsuccessful attempts. We also 
captured NCPACE participant demographics at the time of the first course 


to include the fiscal year of the first course attempt. 


e Using the DMDC accession data, we identified non-prior service Navy 


accessions, capturing the date of accession. 


e Using the DMDC strength data, we captured additional NCPACE 
participant demographic information at the time of the first course and 


identified participants: 
o with 48-month contracts. 
o who continued to serve beyond their 48-month contract. 


o who promoted to E5 before the end of their 48-month contract. 
We chose to evaluate promotions to E5 due to the competitive 
nature of receiving this promotion within the first four years of 


service. 


e Using the DMDC reenlistment and separation data, we captured the first 


reenlistment or separation decision. 


These datasets were merged. In a manner analogous to that in the study of Mehay 
and Pema (2008), we restricted our sample to NCPACE participants who took at least 


one course during their first term, with a successful participant defined as a sailor who 
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successfully completed at least one course in their first term. We defined the first term as 
the period between accession and 48 months thereafter.5 The resulting data set contains 


47,406 observations.® 


We found that those who succeeded in at least one NCPACE course promoted to 
E5 within 48 months of service at higher rates than those who did not (54.2 versus 49.1 
percent; Figure 15). However, it is worth noting that other things such as rating 
vacancies are a major factor of a participant’s promotion potential. We found that 
extension and reenlistment rates do not vary much with respect to NCPACE success 


(Figures 16 and 17). 
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Figure 15. Rate of Promotion to ES by NCPACE Success (FY 1995—FY2003 Accessions). 
Source: NCMIS and DMDC. 


5 We defined the first term in this manner versus using the reported reenlistment and separation action 
dates because over a quarter of the sample had action dates that occurred more than one year after the end 
of their contract obligation. 


6 We included 16,686 sailors who left before serving 36 months of their 48-month contract. It is worth 
noting that this dataset contains 124 observations which report the sailor’s paygrade as E6 or above at the 
time of his or her first course. 
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Figure 16. Rate of Extension by NCPACE Success (FY 1995—FY 2003 Accessions). 
Source: NCMIS and DMDC. 
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Figure 17. Rate of Reenlistment by NCPACE Success (FY 1995—FY2003 Accessions). 
Source: NCMIS and DMDC. 
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HI. REGRESSION ANALYSIS 


A. COURSE COMPLETION MODEL 


The course completion model seeks to identify sailor and course factors 
associated with successful first attempts. We used the data set of first-time NCPACE 


enrollments described in the previous chapter for the course completion model. 


1. Dependent Variable 


The dependent variable for this model is an indicator variable that equals | if the 
course was completed successfully and 0 if the course was not completed successfully. 
Letter grades below “C minus,” non-passing marks in non-letter graded courses, 
incompletes, and withdrawals were defined as unsuccessful attempts. A successful 
course completion is defined as receiving a “C minus” or above in letter graded courses 
or a passing mark in non-letter graded courses. Course records with invalid grades were 


dropped from the data set. 


2. Independent Variables 


The independent variables we chose are heavily based on previous voluntary 


education studies. We considered the following course characteristics: 


e Course delivery method. Garcia and Joy (1998) and Woosely (2009) 
reported differences between traditional classroom (Instructor) and 
distance learning (Technology) methods. Course delivery methods are 


categorized as Instructor or Technology. 


e Course subject. Garcia and Joy (1998) and Woosely (2009) indicate that 
completion rates for mathematics and physical sciences were lower than 
other courses. We used course subjects and titles to categorize courses 
into the following academic disciplines: Humanities and Social Science, 


Business, Mathematics, Physical Sciences, and Other Sciences. 
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Unit vessel type. The study of Garcia and Joy (1998) indicated that 
submariners experienced higher completion rates. We mapped hull 
numbers found in Unit Identification Code titles provided by NCMIS to 
vessel classification information from the Naval Vessel Register and 


created the following unit types: 
o Surface Combat 
« Aircraft Carrier 
"Amphibious Warfare 
* Coastal Defense 
=" Mine Warfare 
= Surface Combatant 
o Submarine 
o Logistics and Support 
= Combat Logistics 
" Mobile Logistics 
= Support 


o Other Unit Type 


Fiscal year of course start date. This was included in order to control for 


changes in NCPACE and Navy policies over time. 


School. Unlike Tuition Assistance, which can be applied towards the 
educational institution of the sailor’s choice, NCPACE courses are offered 
by a more or less fixed set of schools. Of the 19 schools that administered 
NCPACE courses, three schools administered over 90 percent of the 


courses associated with first attempts. We have categorized schools as 
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follows to control for the variation that may occur among them: Central 
Texas College, Coastline Community College, Richland College, and 
Other School. 


We considered the following sailor characteristics in the course completion 


model. Factors that may vary over time were captured around the start of the course: 


AFQT percentile score. AFQT percentile score is a continuous variable 
ranging from 0 to 99 which captures the sailor’s performance on the 


Armed Forces Qualification Test. 


Education. Education reports the level of education attained by the sailor 
and has been used in past studies to capture an individual’s motivation. 
We have chosen the following categories: Did Not Complete High School, 
received an Alternative High School Credential, received a High School 


Diploma, and Completed at Least Some College. 
Rank. Enlisted ranks are treated as categorical variables. 


Rating. Navy enlisted ratings are classified into 23 occupational fields as 
defined by Navy Enlisted Manpower and Personnel Classifications and 
Occupational Standards. We grouped the occupational fields into five 


areas based on functional area: 

o Administrative 
» Administration (LN, NC, PS, RP, YN) 
*" Communications (IT) 
" Logistics (LS, SH) 
=" Media (MC) 
=" Music (MU) 

o First Responder 


" Health Care (HM) 
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= Law Enforcement/Security (MA) 
o Intelligence and Special Warfare 
" Cryptology (CTI, CTM, CTN, CTR, CTT) 
" Intelligence (IS) 
" Special Warfare/Diving (EOD, ND, SB, SO) 
o Maintenance 
« Aviation Ground Support (ABE, ABF, ABH, AS) 


» Aviation Maintenance/Weapons (AD, AE, AM, AME, AO, 
AT, AZ, PR) 


=" Construction (BU, CE, CM, EA, EO, SW, UT) 
= General Seamanship (BM) 
= Marine Engineering (EM, EN, GSE, GSM, IC, MM) 
" Ordnance Systems (GM, MN, MT) 
# Ship Maintenance (DC, HT, MR) 
= Weapons Control (ET, FC, FT) 
o Operations 
» Air Traffic Control (AC) 
» Aviation Sensor Operations (AW) 
" Meteorology and Oceanography (AG) 
= Sensor Operations (STG, STS) 
= Ship Operations (OS, QM) 


Race. We grouped the races into the following categories: White, 
American Indian/Alaska Native, Asian, Black or African American, 


Native Hawaiian or other Pacific Islander, and Other Race. 
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e Ethnicity. We divided this group into Hispanic and non-Hispanic. Sailors 
with an ethnicity of Cuban, Mexican, Puerto Rican, Latin American of 


Hispanic Descent, and Other Hispanic Descent were defined as Hispanic. 
e Sex. We divided this group into Male and Female. 


e Marital status. We divided this group into Married and Not Married. 


3. Model Assumptions, Model, and Model Diagnostics 


Logistic regression models were used to predict outcomes throughout this chapter. 
To predict the successful completion of the first course for participant i, we assume 
Course completion; is a Bernoulli random variable with probability distribution as 


follows: 














Course completion; Probability 
1 P(Course completion; = 1) = 7; 
0 P(Course completion; = 0) = 1 — 7; 
Table 1. Probability Distribution of Course completion; 


We chose the following logistic response function: 


axpta; F) 
E(n) = omprr BD 


where x7; = [ 1, Course delivery method, ..., Marital status; | and B° = [ Bo, fi, ..., B13 |. 


The equivalent logit transformation is In ta = Po + £1 Course delivery method + P2; 
Course subject; + B3; Unit type; + f4 Fiscal year; + Bs; School; + Bs AFOT + fy; 
Education; + Bs; Rank; + Bo; Rating; + Bio; Race; + B11; Ethnicity + B12 Sex + B13 Marital 
status (Montgomery, 2001). The analysis of deviance table corresponding to the course 
completion model demonstrates that each of the selected factors is statistically significant 


at the .05 level (Table 2). 
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Table 2. Course Completion Model Analysis of Deviance Table 














We used the Hosmer-Lemeshow test to assess the goodness-of-fit of the logistic 
regression models in this thesis. This test evaluates subgroups of subjects based on the 
percentile of the estimated probabilities from the model and measures the difference 
between the observed number of occurrences (from the data set) and the expected number 
of occurrences (based on the model) for each subgroup. The test statistic is defined as: 

a ¥ = ny 
k= 


a = Ty} 


where n subjects are divided into g subgroups each of approximate size n,’ = n/g. If g= 
me 
Ys 


100, the Ath group contains subjects in the kth percentile. Then we define 9% as #1 , 


ty e 
bia iad J 


f 
the observed number of completions in the Ath group, and 7x as 1 mk , the expected 


number of completions in that group under the model. The distribution of C is 


approximated by the chi-square distribution with g — 2 degrees of freedom (Hosmer and 


Lemeshow, 1989). 


Because of the size of our data sets, we chose 100 as the value of g. The resulting 
Hosmer-Lemeshow test statistic for the course completion model has a p-value of 0.0098 


(Chi-squared = 133.608, df = 98), and Figure 18 is a plot of the observed versus expected 
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values for the course completion model. Although the p-value of the test statistic is 


slightly smaller than .01, the plot indicates that the model fits the data “reasonably” well. 


% Expected Per Group 


Figure 18. 


Observed vs. Expected 
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Hosmer-Lemeshow Plot of Course Completion Model 


The Receiver Operating Characteristic (ROC) is a function that relates the hit rate 


(sensitivity) to the false-alarm rate (false positive rate). 


The diagonal line where 


sensitivity = false positive rate represents the chance line, that is, the ROC for a 


classification which predicts completion purely at random (Egan, 1975). The area under 


a model’s ROC curve (AUC) suggests that model’s ability to accurately predict the value 


of an observation’s response. Hosmer and Lemeshow’s guidelines for interpreting AUC 


values are in Table 3 (Minitab Technical Support Document). 


Table 3. 

















AUC = 0.5 No discrimination 
0.7 < AUC < 0.8 | Acceptable discrimination 
0.8 < AUC <0.9 | Excellent discrimination 
AUC 2 0.9 Outstanding discrimination 











Guidelines for Interpreting Area Under the ROC Curve. Source: Minitab 
Technical Support Document. 


SI. 


Figure 19 shows the ROC curve for the course completion model. We have 
“acceptable” classification results, as all points of the ROC are above the chance line and 


the area under the curve (AUC) is 0.732. 


ROC 
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False Positive Rate 


Figure 19. ROC Curve of Course Completion Model 


4. Analysis 


The results of the course completion model are presented in Table 4. We reject 
the null model in favor of this model, as our difference-in-deviance test statistic 16646.2 
is greater than 7.43 for any reasonable choice for a. The results of this fixed-effects 


model corroborated the following findings in the study of Garcia and Joy (1998): 


e The predicted success rate increases with rank. E4s were twice as likely to 
successfully complete their first course than Els. E6s were about three 


times more likely to complete than Els. 


e Instructor-based courses have higher predicted success rates. Participants 
who take Instructor PACE courses were about are ten times more likely to 


succeed than those taking Technology PACE courses. 


e Mathematics courses have lower predicted success rates. We found that 


Science courses also have lower predicted success rates. 


e Submarine units have higher predicted success rates. 
a2 


We also observed the following: 


Higher AFQT scores and levels of education were associated with higher 
predicted success rates. Participants who have taken some college courses 
were twice as likely to succeed compared to those who did not finish high 


school. 


Courses sponsored by Coastline Community College have higher 


predicted success rates. 
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Null Deviance: 131532.1 on 101481 degrees of freedom 
Residual Deviance: 114885.9 on 101433 degrees of freedom 
11432 observations deleted due to missing values 





Table 4. Course Completion Model Results 
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B. FIRST COURSE EFFECTS MODELS 


In addition to identifying factors affecting course completions, we attempted to 
estimate the effects of first course results on subsequent attempts. This was done to 
evaluate a failed first course’s potential as a learning experience for the sailor and a 
screening tool for NCPACE. Though passing the first course and passing a subsequent 
course are both correlated with a sailor’s innate ability and motivation to succeed 
academically, passing a subsequent course may also be a function of skills acquired from 
the first course such as effective study habits and time management. Since NCPACE 
participants are working full time aboard ships, sometimes on non-traditional work 
schedules, these skills are especially of valid concern. The outcome of the first course 
can be a valuable tool for both sailors and NCPACE administrators to evaluate whether 
sailors have learned the necessary skills in addition to having the ability and motivation 


to succeed. 


We used the data set of first and second NCPACE course attempts described in 
the previous chapter to evaluate how a first course affects a sailor’s decision to attempt a 
subsequent course and how a first course result affects a sailor’s performance in a 
subsequent course given that the sailor decides to attempt a subsequent course. 


Approximately 45 percent of NCPACE participants have taken more than one course. 


1. Dependent Variables 


We constructed two models to address the questions in the previous chapter: 


e We predicted a sailor’s decision to attempt a subsequent course. For this 
model, the dependent variable is an indicator variable that equals 1 if a 
sailor attempts a second course and 0 if the sailor does not attempt a 


second course. 


e We predicted a sailor’s performance in a second course. For this model, 
the dependent variable is an indicator variable that equals 1 if a sailor 
completes the second course successfully and 0 if a sailor does not. This 


model is restricted to those who taken a second course. 
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2. Independent Variables 


For the two models specified above, we considered the first course performance 
as an independent variable in addition to the variables we chose for the course completion 
model. The course performance variable equals 1 for successfully completed first 
courses and 0 for unsuccessful first courses. The model that evaluates a sailor’s decision 
to take a subsequent course used course and sailor characteristics associated with the first 
course. The model that predicts performance in subsequent courses used course and 
sailor characteristics of the second course, with the exception of race, ethnicity, sex, and 


marital status (which are assumed to remain constant over time). 


3. Model Assumptions, Model, and Model Diagnostics 


To predict a sailor’s decision to attempt a subsequent course, we assume Attempt 
subsequent course; is a Bernoulli random variable with probability distribution as 


follows: 














Attempt subsequent course; Probability 
1 P(Attempt subsequent course; = 1) = 1; 
0 P(Attempt subsequent course; = 0) = 1 — 7; 
Table 5. Probability Distribution of Attempt subsequent course; 
Ts 


We chose the logit transformation In T= # = fo + Bj; Course subject; + B2; Unit 
type; + 3; Fiscal year; + B4 School; + Bs AFOT + Bo; Education; + fh7 Rank; + Bs; Rating; 
+ fo Race; + Bio Ethnicity + Bi, Sex + B12 Successful completion of first course. We 
omitted Course delivery method and Marital status because the analysis of deviance table 
corresponding to the model containing these factors demonstrates that these factors are 


not statistically significant at the .05 level (Table 6). 
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The Hosmer-Lemeshow test statistic has a p-value of 0.0413 (Chi-squared = 
123.5834, df = 98), and Figure 20 is a plot of the observed versus expected values for the 
attempt subsequent course model. Although the p-value of the test statistic is smaller 
than .05, the plot indicates that the model “somewhat” fits the data. Figure 21 shows the 
ROC curve for the attempt subsequent course model. The AUC is 0.654 and all points lie 


above the chance line, indicating that the model predicts better than random guessing. 
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Figure 20. Hosmer-Lemeshow Plot of Attempt Subsequent Course Model 


37 


Sensitivity 


Figure 21. 
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ROC Curve of Attempt Subsequent Course Model 


To predict a sailor’s performance in a subsequent course given that they decide to 


attempt a subsequent course, we assume Complete subsequent course; is a Bernoulli 


random variable with probability distribution as follows: 





Complete subsequent course; 


Probability 








1 P(Complete subsequent course; = | | Attempt 
subsequent course; = 1) = 7; 
0 P(Complete subsequent course; = 0 | Attempt 


subsequent course; = 1) = 1-7; 





Table 7. 


Probability Distribution of Complete subsequent course; 


T 


We used the logit transformation In 1= @ = By + B; Course delivery method + Bj 
Course subject; + B3; Unit type; + f4 Fiscal year; + Bs; School; + Bs AFOT + fyi 
Education; + Bs Rank; + Po; Rating; + B10; Race; + B11 Sex + B12 Successful completion of 


first course. We omitted Ethnicity and Marital status because the analysis of deviance 


table corresponding to the model containing these factors demonstrates that these factors 


are not statistically significant at the .05 level (Table 8). 
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Table 8. Complete Subsequent Course Model Analysis of Deviance Table 


The Hosmer-Lemeshow test statistic has a p-value of 0 (Chi-squared = 167.7942, 
df = 98), and Figure 22 is a plot of the observed versus expected values for the complete 
subsequent course model. Although the p-value of the test statistic is small, the plot 
indicates that the model fits the data “reasonably” well, though the mid-range observed 


values are slightly lower than the expected values. 
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Figure 22. Hosmer-Lemeshow Plot of Complete Subsequent Course Model 
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Figure 23 shows the ROC curve for the complete subsequent course model. We 





have “outstanding” classification results, as the AUC is 0.906. 
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ROC Curve of Complete Subsequent Course Model 


Figure 23. 


4. 


Analysis 
The results of the attempt subsequent course model are presented in Table 9. 
We reject the null model in favor of this model, as our test statistic 7993.7 is greater than 
Passing the first course has the effect of 


047 for any reasonable choice for a. 
multiplying the odds of completing the second course by 2.5. 
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131259.4 on 100984 degrees of freedom 
11412 observations deleted due to missing values 
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Attempt Subsequent Course Model Results 


The results of the complete subsequent course model are presented in Table 10. 
We reject the null model in favor of this model, as our test statistic 63477.18 is greater 
than y°u,47 for any reasonable choice for a. Passing the first course has the strongest 
effect on odds of completing the second course. We also found the following similarities 


to the first course completion model: 
e Higher ranks have higher predicted success rates; 
e Instructor-based courses have higher predicted success rates; 
e Mathematics courses have lower predicted success rates; 
e Submarine units had higher predicted success rates; 


e Higher AFQT scores and levels of education were associated with higher 


predicted success rates; and 


e Courses sponsored by Coastline Community College had higher predicted 


success rates. 
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Complete Subsequent Course Model Results 
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C. PROMOTION MODEL 


The goal of the promotion model is to determine if successfully completing 
NCPACE courses has an effect on promotion. This model considers promotions to E5 
and is restricted to NCPACE participants with four-year contracts in their first term. We 
included 16,686 sailors who left before serving 36 months of their 48-month contract in 


the sample used for the promotion model. 


1. Dependent Variable 


The dependent variable for this model is an indicator variable that equals | if the 


sailor promoted to E5 by the end of their first term and 0 otherwise. 


2s Independent Variables 


The independent variables we chose are identical to the variables chosen for the 
Effects of First Course Results model with the course-specific variables omitted, an 
indicator variable added which captures if the participant succeeded an NCPACE course 
during their first term, and fiscal year indicator variables for the date of accession and 


separation. 


3. Model Assumptions, Model, and Model Diagnostics 


To predict a sailor’s decision promotion to E5, we assume Promoted; is a 


Bernoulli random variable with probability distribution as follows: 














Promoted; Probability 
1 P(Promoted; = 1) = 7; 
0 P(Promoted; = 0) = 1-17; 
Table 11. Probability Distribution of Promoted; 
T 


We used the logit transformation In 1= © = fo + f, Succeeded at least one 
NCPACE Course + $2; Unit type; + £3; Fiscal year of first NCPACE course; + B4 AFOT + 
fs; Education; + fo; Rank; + P7; Rating; + Bs; Race; + Bo Marital status + B19 Fiscal Year 


of Accession + Bi; Fiscal Year of Separation/Reenlistment. We omitted Ethnicity and 
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Sex because the analysis of deviance table corresponding to the model containing these 


factors demonstrates that these factors are not statistically significant at the .1 level 


(Table 12). 
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Promotion Model Analysis of Deviance Table 


Although the Hosmer-Lemeshow test statistic has a p-value of 0 (Chi-squared = 
210.6678, df = 98), the plot indicates that the model fits the data “reasonably” well 
(Figure 24). 
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Figure 24. _ Hosmer-Lemeshow Plot of Promotion Model 
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Figure 25 shows the ROC curve for the promotion model. We have “excellent” 


classification results, as all points of the ROC are above the chance line and the AUC is 


0.840. 
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Figure 25. ROC Curve of Promotion Model 


4. Analysis 


The results of the promotion model are presented in Table 13. We reject the null 
model in favor of this model, as our test statistic 16713.14 is greater than y°.,61 for any 
reasonable choice for a. We found that those who successfully completed at least once 


course are predicted to be more likely to promote to ES by the end of their first term. 
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Succeeded at least one NCPACE Course in our model to see if it is sensitive to sailors 
who attrite early. We reran the regression with a sample that excluded those who left 


before serving 36 months of their 48-month contract. 
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101.63 on 41809 degrees of freedom 


5535 observations deleted due to missing values 


Table 13. 


Promotion Model Results 
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The resulting p-value of this 





variable with the modified sample was 0.04, with a coefficient of .094 (odds ratio of 


1.09). The promotion effects in the model without early separators are slightly smaller. 


D. RETENTION MODELS 


The objective of the extension and reenlistment models are to determine if success 
in NCPACE affects a first term sailor’s decision to extend their contract beyond its 
original term of 48 months. Like the promotion model, the retention models were 
restricted to NCPACE participants with four-year contracts in their first term and 


included sailors who left before serving 36 months of their 48-month contract. 


1. Dependent Variable 


The dependent variable for the extension model is an indicator variable which 


equals 1 if the sailor is still serving after 48 months of service and 0 if the sailor did not. 

The dependent variable for the reenlistment model is an indicator variable which 
equals | if the sailor has a reenlistment action on record after accessing 0 if the sailor 
does not. 

Zi Independent Variables 

The independent variables we chose are identical to the variables chosen for the 
promotion model. 

3. Model Assumptions, Model, and Model Diagnostics 


To predict a sailor’s decision to reenlist, we assume Reenlist; is a Bernoulli 


random variable with probability distribution as follows: 














Reenlist; Probability 
1 P(Reenlist; = 1) = 7; 
0 P(Reenlist; = 0) = 1-17; 
Table 14. Probability Distribution of Reenlist; 


48 


rt 
For the extension model, we used the logit transformation In 1- m™ = fo + f; 


Succeeded at least one NCPACE Course + f2; Unit type; + £3; Fiscal year of first 
NCPACE course; + Bs AFOT + fs; Education; + fo; Rank; + B7 Rating; + Bs Race; + Bo 
Ethnicity + B19 Sex + B11; Marital status + B12 Fiscal Year of Accession + B13 Fiscal Year 
of Separation. The analysis of deviance table corresponding to the course completion 
model demonstrates that the selected factors are statistically significant at the .05 level 


(Table 15). 







































































Df Deviance Resid. Df Resid. Dev | Pr (Chi) 
NULL 41900 55100 0 
NCPACESuccess L 4.88 41900 55100 0.0272 
UnitType 3 211 41900 54900 0 
FY LS 14200 41900 40700 0 
Afgt L 418 41900 40300 0 
Education 3 319 41800 40000 0 
Rank 8 3060 41800 36900 0 
Rating 4 79 41800 36800 3.33E-16 
Race 2 2010 41800 34800 0 
Hispanic L 8.17 41800 34800 0.00426 
Female L 49.3 41800 34800 2.15E-12 
Married L 42.3 41800 34700 7.98E-11 
AccFY 9 1950 41800 32800 0 
SepReenFY 13 3350 41800 29400 0 
Table 15. Extension Model Analysis of Deviance Table 


The Hosmer-Lemeshow test statistic has a p-value of 0 (Chi-squared = 950.4458, 
df = 98), and Figure 26 is a plot of the observed versus expected values for the extension 
model. The plot shows that the model “somewhat” fits the data, though mid-range 
observed values are slightly higher than the expected values and the upper-range 


observed values are slightly lower than the expected values. 
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Observed vs. Expected 
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Figure 26. _Hosmer-Lemeshow Plot of Extension Model 


Figure 27 shows the ROC curve for the extension model. We have “outstanding” 
classification results, as all points of the ROC are above the chance line and the AUC is 


0.917. 
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Figure 27. ROC Curve of Extension Model 


a 
For the reenlistment model, we used the logit transformation In T= F = By + f; 
Succeeded at least one NCPACE Course + fo; Unit type; + $3; Fiscal year of first 


NCPACE course; + fs; Education; + Bo; Rank; + B7; Rating; + Bs; Race; + Bio Sex + B11 
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Marital status + B12 Fiscal Year of Accession + B13 Fiscal Year of Separation. We 


omitted Afgt and Ethnicity because the analysis of deviance table corresponding to the 


model containing these factors demonstrates that these factors are not statistically 


significant at the .05 level (Table 16). 
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Table 16. Reenlistment Model Analysis of Deviance Table 


The Hosmer-Lemeshow test statistic has a p-value of 0 (Chi-squared = 1313.531, 
df = 98), and Figure 28 is a plot of the observed versus expected values for the 


reenlistment model. The plot reveals some systematic under- and over-predicting. 
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Figure 28. Hosmer-Lemeshow Plot of Reenlistment Model 
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Figure 29 shows the ROC curve for the reenlistment model. We have “excellent” 
classification results, as all points of the ROC are above the chance line and the AUC is 


0.876. 
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Figure 29. ROC Curve of Reenlistment Model 


4. Analysis 


The results of the extension model are presented in Table 17. We reject the null 
model in favor of this model, as our test statistic 25699.82 is greater than y°.,63 for any 
reasonable choice for a. We found that completing at least once course does not have a 
Statistically significant effect on extending, as the p-value associated with this factor 


exceeds any reasonable cutoff. 
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5535 observations deleted due to missing values 


Table 17. Extension Model Results 
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The results of the reenlistment model are presented in Table 18. We reject the 
null model in favor of this model, as our test statistic 18459.6 is greater than 7,61 for any 


reasonable choice for a. We found that those who successfully completed at least once 


course are predicted to be slightly less likely to reenlist in the Navy. 
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Table 18. Reenlistment Model Results 
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In a manner similar to the promotion model, we examined the independent 
variable Succeeded at least one NCPACE Course in the reenlistment model to see if it is 
sensitive to sailors who attrite before they are able to make a decision to reenlist. We 
reran the regression with a sample that excluded those who left before serving 36 months 
of their 48-month contract. The resulting p-value of this variable with the modified 
sample was 0, with a coefficient of -.141 (odds ratio of .868). The negative reenlistment 


effects in the model without early separators are slightly larger. 
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IV. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 


1. 


Factors Associated with Successful NCPACE Course Completions 


We evaluated the likelihood of successful first course completions by course and 


sailor attributes. The descriptive statistics and multivariate analysis demonstrated the 


following: 


2. 


Instructor-based courses have higher predicted success rates. We found 
that participants who take Instructor PACE courses were about ten times 


more likely to succeed than those taking Technology PACE courses. 


Success rates also vary across course subjects. Mathematics courses had 


the lowest predicted success rates. 


AFQT scores and levels of education were associated with higher 
predicted success rates. Participants who have taken some college courses 
were twice as likely to succeed compared to those who did not finish high 


school. 


Higher ranked sailors succeeded at higher rates than lower ranked sailors 
E4s were twice as likely to successfully complete their first course 
compared to Els. E6s were about three times more likely to complete 


compared to Els. 


There is some variance in success by unit type. Submarine units have 


higher predicted success rates. 


Courses sponsored by Coastline Community College have higher 


predicted success rates. 


Impact of First Course Results 


We found that the successful completion of the first course is positively 


associated with enrollments in subsequent courses and a factor in successfully completing 


ST 


subsequent ones. Successfully completing the first course had the biggest effect on both 
the odds of attempting a subsequent course and successfully completing the subsequent 
course. We also found that the independent variables which explained first course 


completions also explain subsequent course completions. 


3. Impact of Successful NCPACE Course Completions on Promotion 
and Retention 


We evaluated promotion, extension and reenlistment outcomes for first-term 
NCPACE participants with 48-month contracts. We found that those who successfully 
completed at least once course are predicted to be more likely to promote to ES by the 


end of their first term, even after controlling for factors such as rating. 


We found that completing at least once course does not have a statistically 
significant effect on extensions. However, those who successfully completed at least 


once course are predicted to be slightly less likely to reenlist in the Navy. 


B. RECOMMENDATIONS 


Throughout this thesis, we have identified factors that are associated with 
successfully completing the first NCPACE course and attempting and completing 
subsequent courses. We recommend using this information as a basis for creating 


targeted screening guidelines and tailoring academic counseling efforts. 


NCPACE policies and guidelines continually emphasize the role of academic 
counseling and command leadership in ensuring that participants succeed in the program. 
While the data we were able to obtain explained some of the variance in course 
performance, we were not able to measure the effectiveness of academic counseling and 
command support. We recommend that NCPACE collect, store, and make available the 


following data for future evaluations of the effectiveness of the program: 
e pre-NCPACE assessment test scores 
e the participant’s education plan status at the time of the course including: 


o whether the participant has agreed to an education plan 
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o whether the course taken fulfills a requirement for a degree 
e degree completion information 


e exit survey data after each course to capture the following qualitative 


factors: 
o level of support offered by the command leadership 
© perceived quality of instructor and course materials support 


o reason for completing or withdrawing. 
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